• #rstatsnyc
• DataCommunityDC
• PyData
• SciPy
• The Carpentries
  • Software-Carpentry Data Carpentry

Journey to Open Data Science, Anaconda (Continuum Analytics)

Journey to Open Data Science, Anaconda (Continuum Analytics)

• Reproducible-Science-Curriculum/rr-init
• chendaniely/computational-project-cookie-cutter

cd $1
mkdir doc data src bin results

cd doc
echo "Doc directory with one subdirectory per manuscript" > README
touch .gitkeep

cd ../data
echo "Data directory for storing fixed data sets" > README
touch .gitkeep

cd ../src
echo "src for source code" > README
touch .gitkeep

cd ../bin
echo "bin for compiled binaries or scripts" > README
touch .gitkeep

cd ../results
echo "Results directory for tracking computational experiments peformed on data" > README
touch .gitkeep

echo "Folders created."

cd ..

https://www.bi.vt.edu/sdal/projects/data-science-for-the-public-good-program

Blog post: From VMs to LXC Containers to Docker Containers

https://github.com/bi-sdal/dockerimages
https://github.com/bi-sdal/infrastructure_submodules

• Separate Docker Container for installing R packages (rpkgs)
• This installs R packages into a persistent docker volume
• Everyone has /rpkgs mounted in their rstudio container
• Add /rpkgs to everyone’s .libPaths()

site_path = R.home(component = "home")
fname = file.path(site_path, "etc", "Rprofile.site")
write(".libPaths(c('/rpkgs', .libPaths()))", file = fname, append = TRUE) # prepend to .libPaths
write('local({r <- getOption("repos"); r["CRAN"] <- "https://cloud.r-project.org/"; options(repos=r)})',
      file = fname, append = TRUE)

• In rpkgs: install.packages()

Unless you need system libraries (e.g., CentOS)

https://github.com/bi-sdal/infrastructure_submodules#installing-the-system-library-for-all-docker-images

1. Add the installation in the mro Dockerfile, e.g., yum install -y jq-devel && \
2. Build the base mro, rpkgs, rstudio, and shiny images
3. Push the images to dockerhub, docker push

docker push sdal/mro-c7sd_auth
docker push sdal/rss-mro-c7sd_auth
docker push sdal/rpkgs-mro-c7sd_auth
docker push sdal/shy-mro-c7sd_auth

1. Pull the images from dockerhub, docker pull

docker pull sdal/mro-c7sd_auth
docker pull sdal/rss-mro-c7sd_auth
docker pull sdal/rpkgs-mro-c7sd_auth
docker pull sdal/shy-mro-c7sd_auth

1. Start up the RStudio containers, docker-compose -f rstudio-compose.yml up -d --no-recreate

• Open Source Edition great for individual use
  • Pretty much building the Pro stack…
  • nginx
    • web server: reverse proxy, load balancer, and HTTP cache
• Exploring RStudio Pro… much better suited for parallel projects, groups, and teams (?)

docker-compose.yml/rstudio-compose.yml:

volumes:
  rpkgs:

services:
  rstudio_chend:
    image: sdal/rss-mro-c7sd_auth
    container_name: rstudio_chend
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
      - /etc/group:/etc/group
      - /home:/home
      - rpkgs:/rpkgs
      - checkpoint:/checkpoint
    cap_add:
      - SYS_ADMIN
    ports:
      - 3125:8787

• Collaborative \(\LaTeX\) with real-time rendering… with Git!

• Try it with Docker!
• docker-compose.yml
• ShareLaTeX
• MongoDB
• Redis

• Use rprojects!
  • Open them: rstudioapi::openProject(...)
• here package, “A simple interface to rprojroot”: https://github.com/r-lib/here
  • print(R.utils::sourceDirectory(here::here('shiny', 'functions')))
  • No more if (interactive()){...}
• rprojroot: https://github.com/r-lib/rprojroot

• Save out long calculations into intermediate datasets
• Use base::saveRDS() and base::readRDS()
  • vs base::save() and base::load()

v <- 1:10 # I want to save this...

save(v, file = 'awesome_datascience.RData')
rm(v)
load(file = 'awesome_datascience.RData')
v

##  [1]  1  2  3  4  5  6  7  8  9 10

saveRDS(v, file = 'super_awesome_datascience.RDS')
loaded <- readRDS(file = 'super_awesome_datascience.RDS')
loaded

##  [1]  1  2  3  4  5  6  7  8  9 10

• Hardcoding secrets (e.g., passwords, API keys) in your code
• Reverting lines before a commit
• Sourcing a special ignored file

# uses console or rstudio to do password prompt
getPass::getPass("database username")

.secret_to_keep <- function(user, pass) {
    if (is.null(pass)) {
        pass <- getPass("LDAP Password (the one you use to login to Lightfoot and RStudio):")
    }
    secret_to_keep <- c(password = pass,
                        username = user)
    return(secret_to_keep)
}

setup_user_pass <- function(username = unname(Sys.info()['user']),
                            password = NULL,
                            public_key = '~/.ssh/id_rsa.pub',
                            vault = '/home/sdal/projects/sdal/vault',
                            secret_name = unname(Sys.info()['user']),
                            verbose = FALSE) {
    add_user(username, public_key, vault)
    secret_to_keep <- .secret_to_keep(user = username, pass = password)
    add_secret(secret_name, secret_to_keep, users = username, vault = vault)
}

get_my_password <- function(secret_name = unname(Sys.info()['user']),
                            key = local_key(),
                            vault = '/home/sdal/projects/sdal/vault') {
    return(unname(get_secret(secret_name, key , vault)['password']))
}

I learned from last year, we should all probably just create packages for ourselves/group/lab/company

sdalr, https://github.com/bi-sdal/sdalr/blob/master/R/user_pass.R

RStatsNYC 2016: Data Testing

• Repo: https://github.com/chendaniely/2016-04-08-rstatsnyc_testing
• Video: https://www.youtube.com/watch?v=CAy0udiWwmg

• This presentation is written in it!
• I’ve given Meetup Talks about it: NYC, DC
• Websites, Books, Presentations, Dashboards, Reports…
  • https://rmarkdown.rstudio.com/

• How to do something
• Gallery of plots
• Similar to: http://www.cookbook-r.com/Graphs/

• https://www.youtube.com/watch?v=xAoljeRJ3lU
• “Perceptually uniform”, sequential, works well in black-and-white, Colorblind friendly
• Matlab: parula

viscm, a tool to see how “good” your colormap is: http://bids.github.io/colormap/

• R Package!
  • https://cran.r-project.org/web/packages/viridis/vignettes/intro-to-viridis.html
  • by: Simon Garnier, Noam Ross, Bob Rudis, Marco Sciaini, Cédric Scherer

• Domain specific color maps (Oceanography): https://www.youtube.com/watch?v=XjHzLUnHeM0

• Cookbooks for you own usecases (e.g., GIS)
• Make your own “flight rules”

Flight Rules are the hard-earned body of knowledge recorded in manuals that list, step-by-step, what to do if X occurs, and why. Essentially, they are extremely detailed, scenario-specific standard operating procedures. […]

NASA has been capturing our missteps, disasters and solutions since the early 1960s, when Mercury-era ground teams first started gathering “lessons learned” into a compendium that now lists thousands of problematic situations […] and their solutions.

— Chris Hadfield, An Astronaut’s Guide to Life.

Our planet needs our help, and we need (good) science to fix it.

— Greg Wilson